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GLOBAL MOTION ESTIMATION FOR SPRITE GENERATION 

BACKGROUND OF THE INVENTION 

The present invention relates to sprite generation 
and 3-D motion estimation, and newly-developed 
techniques for global motion estimation. 

1 . Introduction 

A sprite is a still image that represents an entire 
visible scene. A sprite may be represented by two- 
dimensional arrays of luminance, chrominance and 
grayscale alpha data. 

In practice, a sprite is generated from computer- 
animated graphics or a collection of natural images 
taken at several portions of the static background. In 
this sense, a sprite has the same meaning as an image 
mosaic in photogrammetry, image processing and computer 
graphics . 

For example, as shown in FIG. 9, a sprite 950 may 
be formed from successive images 910 and 930. Each 
feature in the sprite 950 is registered with a 
corresponding feature from the input images (e.g., 
frames) 910 and 930. Motion parameters describe the 
warping of the sprite that achieves the closest match 
with the input image feature. The sprite may have to be 
warped differently to each input frame in the sequence, 
e.g., due to camera motion or motion of an object. 

For example, the tree object 912 Tl is based on the 
tree object 912 from the first input image 910. The 
house object 914 lf is based on the house objects 914 and 




914 1 from the first and second input images 910 and 930, 
respectively. The tree object 916' ? is based on the 
tree object 91 6 1 of the second input image 930. 

Moreover, if the sprite is a background object in a 
sequence that is not entirely visible in certain frames 
due to foreground objects or camera motion, such as 
panning, a complete sprite can be obtained by gradually 
updating it over the sequence by collecting the relevant 
pixels of the input frames. 

The sprite has become popular in various 
applications such as video games, interactive 
advertising, virtual travel, etc. A sprite provides an 
efficient way to navigate through a scene. Any portion 
of a scene at any angle can be constructed by 
appropriately warping and cropping the sprite. 
Therefore, it is increasingly used in advertisements on 
the Internet. For example, a sprite can be generated 
for the inside of an automobile and a user can rotate 
the viewing angle to look around and/or zoom in or out. 

Another example is sending a stock ticker using a 
sprite rather than moving video, which can reduce the 
amount of data by a factor of more than twenty, 
according to our studies. A logo (e.g., station 
identifier, advertisement or the like) can also be sent 
as a sprite. 

A sprite is very beneficial in terms of coding 
efficiency since there is a redundant or very easy to 
re-encode part of the scene in the sprite that is 
revisited in the new image. Therefore, MPEG-4 (ISO/IEC 
14496-2 Final Draft of International Standard, 
"Information Technology - Coding of audio-visual 
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objects, Part 2 : visual," MPEG98/N2502, Dec. 1998) has 
standardized the sprite coding tools by including them 
in its visual part. A static sprite only provides the 
warping result of the sprite piece, while a dynamic 
5 sprite (DS) and global motion compensation (GMC) allow 
the warping result to blend and locally adjust with the 
previously decoded video object plane (VOP) . A sprite 
piece is basically an image, but has its origin from the 
image mosaic (e.g., mosaic piece). 

10 The static sprite has already been adopted for 

MPEG-4 visual version 1, and the DS and GMC are expected 
to be included in version 2. According to the version 1 
syntax, a static sprite piece is coded as a regular 
intra VOP at once at the beginning of a sequence, or 

15 piece-by-piece in the low latency mode. Global motion 
parameters (up to eight for the perspective motion 
model) are also coded at every VOP. The sprite piece 
and global motion parameter, however, are not part of 
the MPEG-4 encoder and are required as an input to the 

20 MPEG-4 encoder. 

Although MPEG-4 video offers coding tools for 
sprites, its does not specify normatively how to 
generate sprites. Only a brief description of the 
reference algorithm is provided in the informative 

25 section of the MPEG-4 video verification model version 
9.1 (informative section) (ISO/IEC 14496-2 MPEG-4 Video 
Verification Model Version 9.1, MPEG98/M3100, Feb. 
1998) . 

Accordingly, it would be desirable to provide an 
30 improved global motion estimation technique for an off- 
line sprite generation. The technique should extend the 
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reference algorithm developed by core experiments in 
MPEG-4 (refer to MPEG98/M31002 , and J. Konrad and F. 
Dufaux, "Improved global motion estimation for N3," 
ISO/IEC JTC1/SC29/WG11 MPEG98/M3096, Feb. 1998) . 

The original MPEG-4 method uses an iterative 
nonlinear minimization algorithm for the perspective 
global motion estimation. Such an iterative algorithm 
relies heavily on the quality of the initial estimation. 

Accordingly, it would be desirable to improve the 
performance of sprite coding and allow more flexibility 
to the source sequences, e.g., by providing new tools 
such as automatic sprite generation system, first-order 
prediction for initial estimation, delayed elimination 
for outlier rejection, and field based sprite generation 
for an interlaced source. 

It would be desirable to provide a technique for 
determining a more accurate initial estimation based on 
the pre-warping and recursive techniques. 

It would be desirable to improve the accuracy of 
sprite generation and motion parameters with an 
effective outlier rejection scheme. 

It would also be desirable to improve the quality 
of sprite generation and motion parameters under both 
controlled and uncontrolled conditions. The controlled 
condition is the ideal condition that is expected in the 
normal environment, e.g., no noise in the scene, video 
moves slowly from frame to frame. 

The present invention provides a system having the 
above and other advantages . 




SUMMARY OF THE -INVENTION 



The present invention relates to sprite generation 
and 3-D motion estimation, and newly-developed 
techniques for global motion estimation. 

The sprite generation algorithm suggested by MPEG-4 
is not ready for practical use since, in part, its 
global motion estimation technique is susceptible to 
outlier and fast motion problems. 

The present invention addresses these and other 
concerns by providing a complete automatic sprite 
generation system, first-order prediction for an initial 
estimation, delayed elimination for outlier rejection, 
and field-based sprite generation for an interlaced 
source . 

Optionally, higher-order prediction for the initial 
estimation is used to handle more complicated motion. 

The invention is useful for generating sprites, 
e.g., for 3D sequences, stock tickers, interactive 
advertising and a variety of other uses. 

In particular, automatic sprite generation is 
provided by performing shot detection on the input 
images to provide a group of successive images that 
share a common static background for use in forming a 
sprite. The shot detection may detect panning, in which 
case block matching in the initial estimation of a 
motion parameter is turned on, or zooming, in which case 
block matching in the initial estimation is turned off. 

The initial estimation of motion parameter data for 
forming the sprite from a series of input images may be 
provided based on motion parameters of at least two 
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input images that precede the initial input image, e.g., 
using linear prediction or non-linear prediction of 
motion parameters of the at least two previous input 
images. In a particular embodiment, the initial 
estimation of the motion parameter S n is provided 
according to: S n =S n _i + (S n -i - S n - 2 ) t where S n -i is a motion 
parameter of an input image that precedes the initial 
input image, and S n _ 2 is a motion parameter of the next 
preceding input image . 

In a further embodiment, delayed outlier rejection 
is achieved by providing at least first and second 
successive estimates of motion parameter data for 
providing a registration between the sprite and the 
initial input image. At least first and second errors 
between pixels of the initial input image and pixels of 
the sprite are determined based on the at least first 
and second estimates, respectively. The second estimate 
is obtained by updating the first estimate based on the 
first error. The pixels whose second error exceeds 
their first error are eliminated from consideration in 
generating the sprite. 

In a further embodiment, a method for generating 
sprite data from a series of input images that comprises 
interlaced first and second fields, includes the steps 
of: generating separate first and second field sprites 
with separate sets of motion parameter data for the 
first and second fields of the input images, 
respectively, and encoding the first field sprite and 
its associated motion parameter data separately from the 
second field sprite and its associated motion parameter 
data for transmission to at least one decoder. 




A corresponding method for processing sprite data 
at a decoder includes the step of: receiving data 
transmitted from an encoder, where the data includes an 
encoded first field sprite and its associated motion 
parameter data, and a separately encoded second field 
sprite and its associated motion parameter data. The 
first and second field sprites are generated from 
interlaced first and second fields, respectively, of at 
least one initial input image at the encoder. The 
method includes the further steps of: decoding the 
encoded first field sprite and its associated motion 
parameter data to provide a first field sprite output; 
decoding the encoded second field sprite and its 
associated motion parameter data to provide a second 
field sprite output; and combining the first field 
sprite output and second field sprite output to form a 
combined sprite output, e.g., for a display such as a 
television or computer monitor. 

Corresponding methods and apparatuses are 
presented. 



8 



BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 illustrates a sprite generation apparatus in 
accordance with the present invention. 

FIG. 2 illustrates a flow chart of a global motion 
estimation method. 

FIG. 3 illustrates an automatic sprite generation 
system in accordance with the present invention. 

FIG. 4 illustrates an example of an outlier. 

FIG. 5(a) illustrates a misaligned image that 
results with the Truncated Quadratic (TQ) outlier 
rejection method. 

FIG. 5(b) illustrates an outlier that results with 
the Truncated Quadratic (TQ) outlier rejection method. 

FIG. 6 illustrates a delayed elimination algorithm 
in accordance with the present invention. 

FIG. 7 illustrates a method for global motion 
estimation in accordance with the present invention. 

FIG. 8 illustrates a sprite decoder. 

FIG. 9 illustrates the formation of a sprite. 

FIG. 10 illustrates an encoder and decoder for 
processing an interlaced source in accordance with the 
present invention . 



DETAILED DESCRIPTION OF THE INVENTION 



The present invention relates to generating sprite 
images . 

2. Sprite generation and global motion estimation 

This section provides an overview of the sprite 
generation for the MPEG-4 encoder. The required inputs 
to the MPEG-4 encoder are the sprite piece and the 
motion parameter. This motion parameter includes a 
traditional motion vector (as in MPEG-2) as a special 
case (for the translation motion model) . The MPEG-4 
encoder compresses the sprite piece using an intra 
coding mode. The image at any time instance is 
represented by sprite trajectories (sprite reference 
points) which are the 1, 2, 3, or 4 corners (depending 
on the motion model) of the warped sprite piece 
according to the motion parameter. 

The sprite piece and the motion parameter are 
closely related and, hence, are determined 
simultaneously in the sprite generation. The sprite 
piece is built by warping the new image according to the 
global motion parameter and updating it on the current 
sprite based on the blending scheme. A key problem is 
how to find the motion parameter for every image. This 
problem falls in the general case of an image 
registration problem. The optimal solution is the 
motion (warping) parameter that maximizes the 
correlation between the registering (new image) and the 
registered (sprite) images. A statistical approach for 
image registration with only translational motion is 
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provided in W. K. Pratt, "Correlation techniques of 
image registration," IEEE Trans. Aerospace Electronics 
and Systems, vol. AES-10, pp. 353-358, May 1974. 

Three-dimensional (3D) motion, which requires a 
5 complete geometry transformation in 3D space, allows a 
full range of motion, e.g., rotation, zoom, and shear 
between two images. An overview of the 3D motion 
estimation methods is given in T . S. Huang, and A. N. 
Netravali, "Motion and structure from feature 
10 correspondences: A review," Proc. IEEE, vol. 82, pp. 
252-268, July 1997. 

For natural video, MPEG-4 sprite coding is 
appropriate for an image sequence containing a static 
background and experiencing a camera motion. This 
15 situation is valid for MPEG-4 since coding of arbitrary 
shapes allows the input to be separated into several 
objects depending on the associated alpha plane and, 
hence, sometimes the moving part can be separated into 
another object and be excluded from the sprite. Under 
20 this assumption, part of a new image experiences a 

deformation from part of the previous image or sprite, 
and pixels in a sprite are hence related to the pixels 
in a masked image by the planar perspective 
transformation (forward warping), discussed in 
25 MPEG98/M3100 and H. Y. Shum and R. Szeliski, "Panoramic 
image mosaics," Technical Report, MSR-TR-97-23, 
Microsoft Research, 1997, as follows: 

t nj0]x + m[l]y + m[2] 
~ /^6]x + /w[7])/ + /w[8] 
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n£3]x + m[4]y + m[5] 
/^6]x + /w[7]y + /w[8] 
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where m[0] to m[8] are perspective coefficients (m[8] is 
set to 1 for the planar case), and (x,y) and (x',y') are 
the coordinates of the pixels in the warped image and 
sprite, respectively. The inverse warping describes the 
relation between the pixels in the masked image and the 
pixels in the sprite, i.e., 

(m[4]m[S] - m[5]m[7])x*+(m[2]m[7] - m[\]m[SJ)y+m[\]m[5] - m[2]m[4] 
X ~ (/w[3]m[7] - m[4}m[6])x l +(m[l]m[6] - m[0]m[7])y+m[0]m[4] ~ m[l]m[3] 

(/^5M6]-/^3>^ 
y ~ (m[3M7]-^4M6]^ 

Perspective coefficients in the planar perspective 
transformation are parameters that describe the 
deformation between the masked image and the sprite. 
Global motion estimation attempts to find the best 
registration (motion parameter) of the masked image on 
the sprite which, in turn, determines the deformation 
and the perspective coefficients. Once the masked image 
is transformed via the inverse warping, it is blended to 
the current sprite to build an updated sprite. Inverse 
warping is normally used instead of forward warping to 
avoid the hole in the updated sprite caused by the 
stretching of the current sprite in the forward warping. 
The invention is applicable to either forward or reverse 
warping . 

Blending is the final process of off-line sprite 
generation. Blending takes two inputs and blends them 
together according to a weighting factor. A weighting 
factor specifies the mixing ratio between the two inputs 
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at any position. In MPEG-4, it is a integer variable 
for any pixel in a sprite, and it is increased by a 
defined constant such as one whenever its corresponding 
pixel is updated. By incrementally adding (updating) 
5 the new image to the sprite with the equal weight, noise 
in a single image is smoothened out. Note that blending 
in the on-line sprite coding process constructs an 
updated sprite using the new VOP and the sprite. In 
*i this case, the blending factor is a floating point 

Jj 10 constant between zero and one (the blending factor is 

always one for a static sprite) . 
% i 3. MPEG-4 perspective motion estimation algorithm 

q\ MPEG-4 activities on sprite generation are 

^" conducted within the core experiment, N3 . Two global 

rn 15 motion estimation algorithms (MPEG98/M3096 and C. Gu, T. 

fj Markoc, R. Szeliski and M. C. Lee, "Results report on 

LI 

fr> core experiment N3 - Sprite generation, " ISO/IEC 

P JTC1/SC29/WG11 MPEG96/M1350, Sep. 1996) have been 

O 

investigated in N3 and widely used in the core 
20 experiment on sprite coding. The M1350 proposal has 

formed a reference algorithm in the annex of the MPEG-4 

VM (M3100) . The M3096 proposal is a refined method 

based on M3100. The following subsections describe 

details of these two algorithms. 
25 3.1 MPEG96/M1350 proposal 

FIG. 1 shows an overview of an off-line sprite 

generation apparatus 100 in accordance with the present 

invention. 

The apparatus 100 is based on the MPEG-4 VM, which 
30 is based on M1350, but proposes a new algorithm for the 

\3 
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motion estimation function 110. All elements in FIG. 1 
are needed to generate the sprite piece. The motion 
estimation function 110 receives mask data, VOP data, 
and data from a frame memory 140. A warping function 
5 120 provides warping based on the VOP data and the 
output of the motion estimation function 110. A 
blending function 130 provides blending based on the VOP 
data and the output of the warping function and the 
frame memory to provide a sprite. In particular, the 
10 blending function 130 updates the sprite piece by 
W blending the current VOP (after an appropriate warping) 

01 

to the existing sprite piece. 
& Similar warping and blending as discussed in the 

L previous section are used in this process. Since the 

^ 15 perspective motion model adopted in MPEG-4 is nonlinear, 

jfn motion estimation, which attempts to minimize the error 

O between the input image and sprite by finding the 

ih optimum deformation parameter of the input image, has to 

W rely on a nonlinear minimization technique. The core 

20 process of perspective motion estimation in MPEG-4 VM is 
a nonlinear minimization technique called "Levenberg- 
Marquardt" (LM) (see W. H. Press, S. A. Teukolsky, W. T. 
Vetterling and B. P. Flannery, "Numerical recipes in C, " 
Cambridge Univ. Press, 1992) which is essentially an 
25 iterative algorithm. The LM technique solves the 

nonlinear least-square problem efficiently by adaptively 
adjusting its algorithm between the steepest descent and 
the inverse Hessian method according to the % 2 merit 
function (although % 2 is not used in VM) . LM tries to 
30 solve the normal equation. 
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where A is an approximate Hessian matrix which has the 
component 



jb is the weighted gradient vector which has the 
component 



A Gaussian pyramid can be used together with LM to 
speed up the process. Motion parameters between the 
pyramid level are related by 

m 6 ~m e 1 2,/w 7 -m 1 1 2,/w 2 = m 2 * 2,m 5 = m 5 * 2 . 

3.2 MPEG96/M1350 proposal 

FIG. 2 illustrates an improvement to the MPEG-4 VM 
perspective motion estimation. 

This technique starts with an initial estimation 
using a three-step search block matching algorithm 
(block 210) to estimate the translational motion 
parameter. This estimation is performed at the coarsest 
(lowest) resolution. The motion parameter obtained from 
the initial estimation starts the LM algorithm, which is 
performed recursively starting from the coarse 
resolution and refining the result at the finer (higher) 
resolution (blocks 220, 280) . The result of every stage 
is the result after 32 iterations (blocks 250-270) . 
However, the iteration stops once the updated parameter 
is smaller than a preset threshold (block 240) . 

The effect of an outlier is alleviated by an 
outlier rejection method such as "Truncated Quadratic" 
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(TQ) (block 230) . It aligns the input image and sprite 
together according to the motion parameter determined by 
LM of that stage and then calculates an error (a 
quadratic function in this case) of every pixel inside 
5 both objects. Any pixel which contributes to the 10% 
highest error of that image is excluded from the 
subsequent minimization. The truncated quadratic 
process is carried out at the first iteration of every 
resolution . 

yfj 10 4. The proposed modification techniques 

3 ill 

In accordance with the invention, this section 

%l describes the modification of the perspective motion 

!S? estimation to improve the performance in an uncontrolled 

M* condition. Such a condition is likely to occur when 

JL 15 there is a high motion in the scene such as a fast pan 

01 of a camera. The proposed modification techniques use 

5? the LM to minimize the differential frame difference 

yl 

Q between the sprite and input image. A high accuracy of 

M the initial estimation is necessary for the LM to 

20 converge to the global minimum. The first subsection 
(4.1) will discuss this problem in more detail. 

A modification to the MPEG-4 VM to improve the 
accuracy of the initial estimation will also be 
described. The second subsection (4.2) also attacks 
25 another problem in an uncontrolled condition where noise 
exists in the scene. The second subsection will show 
that the TQ method is not suitable for outlier 
rejection, and a better alternative is proposed. The 
third subsection (4.3) will discuss a method to handle 
30 interlaced sources. This problem has not yet been 
investigated in the core experiment N3. 



On-line sprite generation from a natural video 
sequence requires a preprocessing stage that can create 
the input sequence suitable for a sprite automatically. 

FIG. 3 shows an overview of the automatic sprite 
generation system of the present invention. This system 
includes shot detection (block 310) to group images that 
share the same static background and are thus suitable 
for sprite generation. A shot is a sequence of images 
covering the same scene (not necessarily static) from 
one camera. A sprite piece is automatically updated 
every time that the system receives a suitable input 
image. The sprite is useable when there is enough data 
in the same scene; otherwise, this sequence is 
considered not suitable for sprite coding. A good shot 
detection algorithm should be able to specify the 
characteristic of motion (e.g., zooming, panning) in 
that shot so that the global motion estimation can 
adjust its parameters properly (e.g., turn on/off block 
matching in the initial estimation when there is a 
pan/ zoom motion) . Any known shot detection scheme may 
be used. 

An automatic masking process (block 310) may be 
required when a foreground object (e.g., an object that 
moves differently from the background) exists in the 
scene. A mask can be synthesized from the color 
distance difference between the foreground and 
background (assuming that color in the foreground and 
background are significantly different) , similar to the 
blue screen technique (see commonly-assigned, co-pending 
U.S. Patent Application No. 09/156,790 to K. Panusopone 
and X. Chen, filed Sep. 17, 1998, and entitled "Modified 
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Chroma Keyed Technique For Simple Shape Coding for 
Digital Video") . In such a case, the whole background 
that can cover the foreground is needed to be taken a 
pri ori . 

Note that the useable data is used by both blocks 
310 and 340, FIG. 3 only considers one shot at a time 
so the shot detection only detects the same shot with 
the useable data (all the pictures in the same shot are 
useable data) . 

The same shot decision block 320 analyzes the input 
frame with the knowledge of the previous useable data to 
decide whether the input frame belongs to the same shot. 
This analysis includes calculating the correlation 
parameter. 

An "enough data" decision block 350 keeps track of 
the duration of the video sequence. The useful input 
for a sprite should be long enough to develop a 
significant trend in the sprite. 

The sprite generation block 340 is covered by FIG. 

1. 

4.1 Initial estimation 

MPEG-4 VM perspective motion estimation works under 
the assumption that there is a small deformation among 
consecutive images. In this case, the LM converges 
reasonably well with the motion parameter of the most 
recent image as an initial estimation. To maintain this 
assumption, the input sequence is required to have only 
a slow motion. However, this causes a significant 
inconvenience to the cameraman. Our experiments confirm 
that reusing the previous motion parameter does not lead 
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to a satisfactory result using LM when the final 
parameter is substantially different from the initial 
estimation. The solution to this problem is to use the 
past motion history to determine the initial estimation. 
5 With this scheme, the initial estimation can be close to 
the final parameter as long as the motion occurs in the 
same pattern no matter how different the parameter is 
between the consecutive images. 

Any kind of (autocorrelation-based) linear 
^ 10 prediction (e.g., differential pulse code modulation - 

y] DPCM) or nonlinear prediction (e.g., spline 

!ifl interpolation) can be used to determine the initial 

4i estimation. The simplest, second order prediction is 

^ described mathematically as follows: 

* 15 S =S . +(£ . -S ,) 

E Where S n = \m[0] m[l] m[2] m[3] m[4] m[5] m[6] m[7] m[S]] 

8] is the motion parameter of the n-th image, and the 

F= superscript "t" denotes a transpose. This second order 

prediction differs from reusing the past motion 
20 parameter by the addition of the residual between motion 
parameters of the past and the next-to-past images. The 
second order prediction works under the assumption that 
the camera operates at more or less the same speed. The 
drawback of this prediction occurs in unusual cases when 
25 the motion does not follow the pattern of the previous 
two images, in which case the predicted initial 
estimation may be worse than that obtained by reusing 
the motion parameter. 

Block matching, e.g., as used in M3096, is not 
30 suitable for initial estimation since it may select a 
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totally irrelevant initial estimation when the 
consecutive pair of images have a strong motion in other 
parameters than' translation, e.g., in case of zooming. 
However, block matching is recommended when the input 
5 sequence experiences an unpredictable pan, such as in 
sports sequence. 

4.2 Outlier Rejection 

It is extremely difficult to guarantee that all 
background in all natural input images are static. An 
£f 10 outlier, which may happen because a leaf on a tree 

*„ s"S 
Tic 

y] moves, or people turn their heads during the shot, 

jp r : causes a contradictory contribution for the LM process. 

*0 The impact of an outlier may be severe if the outlier is 

JV dominant in the scene, e.g., the error from the outlier 

s 15 is particularly strong compared to the error from the 

rest of the scene. LM performance improves dramatically 
□ with outlier rejection because the error from the 

outlier is excluded. Moreover, the outlier provides a 
large error when the motion parameter is almost optimum. 
20 FIG. 4 illustrates an example of an outlier. For 

example, two identical facial images are shown at 400 
and 450, except in image 1 (400), the eyes 410, 415 are 
closed, while in image 2 (450), the eyes 410', 415 f are 
open. The open eyes 410 ! , 415' are thus outliers when 
25 the motion parameter is optimum. 

TQ works on the above principle to selectively 
eliminate all pixels that provide the highest 10 percent 
of errors. That is, according to FIG. 2, the current 
VOP is warped by the warping parameter obtained from the 
30 previous VOP (for the coarsest resolution) or the 

previous stage (coarser resolution) . The errors of all 
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pixels in the current VOP are then sorted, and pixels 
that yield the top 10 percent of the errors are declared 
outliers . 

However, this straightforward technique performs 
poorly when the motion parameter is not close to 
optimum. The problem with TQ in such a case is that an 
important feature in the scene can misalign and act like 
an outlier. 

The present invention overcomes this problem by 
using delayed elimination for outlier rejection. 

FIG. 5(a) and 5(b) show an example of the problem 
with the truncated quadratic method. For example, 
suppose we have two successive frames 510, 520 of the 
symbol X T' 515, 525, respectively, that are misaligned 
as shown in FIG. 5(a). The image 550 of FIG. 5(b) shows 
the outlier in the case that the warping parameter is 
zero (assuming that no previous information is 
available) . Here, unfortunately, all details in FIG. 
5(b) will be eliminated according to TQ and, hence, the 
remaining details won' t be able to determine the correct 
global motion. 

Generally, the outlier pixels behave more randomly, 
like noise, which means they may match worse with the 
better motion parameter while it should not happen in an 
important feature. That is, the inconsistency should 
not happen in an important feature, e.g., a line, which 
usually yields a lower error with a better match (more 
iterations) . It is important to realize that only a few 
crucial features are sufficient to determine the optimal 
motion parameter. In practice, as much test data as 
possible is used in the process because crucial features 
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are very difficult to identify. Test data contains not 
only crucial features and outliers, but also meaningless 
data such as a plain background, which always matches 
well without revealing any information about the optimal 
5 motion parameter. It is likely that the meaningless 
data will dominate in the remainder of the test data 
when only a small amount of error is allowed. The final 
result in such a case will not resemble the optimal 
motion parameter. A better result can be obtained when 
10 more error is allowed and the ratio of the meaningless 
CJ data to the important feature is more balanced. Of 

g*j course, the performance will start to decline if the 

% 4 amount of error is too high and the outlier becomes a 

factor. 

H 15 FIG. 6 illustrates a delayed elimination process in 

n accordance with the invention. Here, the performance of 

0; the outlier rejection can be improved with the addition 

q\ of a delayed elimination process. In particular, 

W instead of processing the error in one pass, more 

20 reliable information is available with a two-pass (or 
more) process. 

In a first iteration (610), the LM minimization 
technique is performed, and an error (el) is calculated 
for every pixel. At block 620, pixels that provide more 
25 error than the preset threshold are marked as probable 
outliers. These pixels are checked again after another 
iteration (630) to allow them to align better if they 
are an important feature. At block 640, pixels that 
have the higher error in the second pass behave like 
30 noise and are declared outliers. At block 650, the 
outliers are eliminated. 
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Note that other configurations based on this 
concept can be implemented with more complexity, such as 
delayed elimination based on three or more stages, 
advance methods to specify an outlier, etc. With three 
or more update stages, there is a trade-off between 
performance and complexity. 

4.3 Sprite generation for an interlaced source 

Currently, MPEG-4 only processes sprite pieces in a 
frame format. There are two possible ways to code the 
interlaced input image as a sprite under the current 
MPEG-4 syntax. The first method forms a single frame 
input by deinterlacing the source and carries on the 
normal sprite generation. The second method forms two 
sprites based on each field then combines both field 
sprites to generate a frame sprite. These two solutions 
have one sprite piece to encode and one set of 
parameters to transmit. However, these approaches do 
not result in optimal performance in many cases. 

FIG. 10 illustrates an encoder 1000 and decoder 
1050 (e.g., in a decoder population) for processing an 
interlaced source in accordance with the present 
invention. Here, the MPEG-4 syntax is modified to 
process an interlaced source as two independent inputs 
(one for each field), e.g., at sprite generation and 
encoding functions 1010 and 1030. After being encoded, 
multiplexed at a mux 1020, and transmitted across a 
network 1040, the two field sprites and associated 
motion parameter data are demultiplexed at a demux 1060, 
and decoded at decoding functions 1070 and 1080. 

The field sprites are output from the respective 
functions 1070 and 1080 and combined at a presentation 
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engine 1090 to form a combined sprite output, e.g., for 
a display 1095. 

Note that the encoder 1000 and decoder 1050 are 
shown in a simplified form. Moreover, while the 
5 functions 1010 and 1030, and 1070 and 1080 are 

illustrated as being separate, they may in practice 
utilize common software and/or hardware as will be 
apparent to those skilled in the art. 

In this embodiment, the two sprite pieces are 
10 encoded separately, and two sets of motion parameters 
have to be transmitted and decoded. However, the 
performance of this sprite is better than the other two 
MPEG-4 compliance methods, e.g., for complex motion, 
where two fields in the same frame are likely to have a 
15 different deformation (motion parameter) . 
Li 5. Sprite Generation Guidelines 

01 Several observations have been collected from our 

p experiment on sprite generation. These observations are 

O guidelines for successful sprite generation. 

20 - 1. The initial estimation is the most 

important factor for the minimization process. A better 
initial estimation has a higher chance of obtaining an 
optimum sprite motion parameter. 

2. The speed of the camera should be constant 
25 (at any speed) because our initial estimation is based 

on a linear prediction model (distance residual) . 

3. A more complicated prediction model (e.g., 
velocity residual) is possible to compensate for the 
highly complex function of camera speed. 

30 4. It is recommended that the change of speed 

or direction of the camera should be done at a gradual 
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rate (e.g., from the rest position to the operating 
speed) since the prediction model needs to fill the 
relevant past history before it can predict a reliable 
initial estimation. 
5 5. Initial estimation based on block matching 

(e.g., M3096) performs poorly when the current image 
experiences zooming, shear or rotation, but it usually 
helps in the case of a complicated pan. Shear is an 
effect of changing a rectangle into a parallelogram. 
10 6. Outlier rejection helps preserve the 

gp important features in the scene. It is very difficult 

j„; to align a fine detail (e.g., text) without reliable 

VJ outlier rejection. 

™;f 7. A two-pass system, which uses a final 

§=& 15 motion parameter of the first pass as an initial 

!L estimation of the second pass, yields a slight 

Ql improvement (less than 1 dB, PSNR) . A higher order 

system (e.g., three, four) is expected to yield a 
p negligible improvement over a two-pass system. 

20 8. Errors in a sprite generated by 

misregistration of any input image have an equal effect 
regardless of the order in which the errors occur. This 
is also true in the more general term, e.g., order of 
the occurrence (VOP number) . 
25 FIG. 7 shows a global motion estimation process in 

accordance with the present invention. 

The process is based on FIG. 2 but differs in 
blocks 710, 715 and 720. 

Specifically, at block 710, an improved initial 
30 estimation of a motion parameter is obtained. Linear 
prediction using eight motion parameters (e.g., m[0] 
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through m[8], discussed previously), or block matching 
may be used. Alternatively, non-linear prediction may 
be used. Moreover, these motion parameters are not part 
of the sprite coding in MPEG-4, which uses the position 
5 of the four corners which are derived from motion 
parameters . 

At block 715, the first two iterations of the 
sprite are performed at a higher pixel resolution. 

At block 600, the delayed outlier rejection of FIG. 
10 6 is performed. 

FIG. 8 illustrates a sprite decoder 800, which 
includes a shape/texture decoder 810 for I-VOPs, a 
shape/texture decoder 830 and a warping vector decoder 
for S-VOPs (sprite VOPs) , a sprite buffer 820, and a 
15 warping function 850 for providing reconstructed 
samples . 

6. Simulation Results 

Parameters in the proposed system that are allowed 
to vary are the search range for block matching, and the 

20 threshold for error truncation. Other variables are 
held constant, i.e., the number of iterations, the 
stopping threshold, the amount/level of pixel resolution 
(e.g., 128x128, 256x256 and 512x512), to draw a clear 
comparison between the proposed system and M3096. All 

25 input sequences are either cropped or masked to exclude 
the foreground object. An uncontrolled condition is 
simulated by skipping a constant number of images to 
create a more severe deformation. 

Tests were run for a variety of targeted 

30 applications (e.g., stock ticker, currency ticker) to 
obtain the final sprite piece generated by the present 
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invention. The sprites were also encoded together with 
their associated motion parameter by the MPEG-4 encoder 
to illustrate the coding efficiency of the proposed 
system. These tests verified the success of the present 
5 invention. 

7 . Conclusions 

This memo describes improvement techniques for the 
MPEG-4 based global motion estimation for sprite 
generation. The proposed techniques perform much better 

10 than the current MPEG-4 scheme in uncontrolled 

situations, which are more likely to occur in practice. 
Several key solutions to the VM global motion estimation 
have been proposed, i.e., initial estimation techniques, 
delayed elimination for outlier rejection, and sprite 

15 generation for interlaced sources. The proposed 
techniques support the current MPEG-4 encoder 
requirement, and are also capable of providing a 
reliable motion parameter for GMC and DS in MPEG-4 
version 2. 

20 Although the invention has been described in 

connection with various specific implementations, it 
should be appreciated that various adaptations and 
modifications can be made thereto without departing from 
the scope of the invention as set forth in the claims. 
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